Shannon Meets Carnot: Generalized Second Thermodynamic Law 
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The classical thermodynamic laws fail to capture the behavior of systems with energy Hamilto- 
nian which is an explicit function of the temperature. Such Hamiltonian arises, for example, in 
modeling information processing systems, like communication channels, as thermal systems. Here 
we generalize the second thermodynamic law to encompass systems with temperature-dependent 
energy levels, dQ = TdS+ < dS/dT > dT, where < • > denotes averaging over the Boltzmann dis- 
tribution and reveal a new definition to the basic notion of temperature. This generalization enables 
to express, for instance, the mutual information of the Gaussian channel as a consequence of the 
fundamental laws of nature - the laws of thermodynamics. 
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The current scientific conception is that the theory of 
information is a creature of mathematics and has its own 
vitality independent of the physical laws of nature. 

The laws of thermodynamics are fundamental laws of 
nature, and in particular, the second thermodynamic law 
linearly relates the change in the entropy, dS, to the 
amount of heat, dQ, absorbed by a system at equilib- 
rium, dQ = TdS, thus defining the temperature, T, of 
the system [l[ . Thermodynamics is primarily an intellec- 
tual achievement of the 19 th century. The first analysis 
of heat engines was given by the French engineer Sadi 
Carnot in his seminal 1824 publication, 'on the Motive 
Power of Fire and on Machines Fitted to Develop that 
Power' [2, 0], laying the foundations to the second law 
of thermodynamics. This paper marks the start of ther- 
modynamics as a modern science which was subse- 
quently evolved to the more general discipline of statis- 
tical mechanics 

Information theory is a statistical theory dealing with 
the limits and efficiency of informatics 0. This theory is 
one of the major enablers of modern technologies in the 
information age, from compressed ZIP files, CDs, MP3s, 
DSL high-speed modems and mobile phones to the Voy- 
ager missions to deep space. Similar to the key role of 
Carnot's paper in the development of thermodynamics, 
the birth of information theory as an independent disci- 
pline is attributed to the landmark publication of Claude 
Shannon in 1948 l A Mathematical Theory of Communi- 
cation' 

The generic problem in information processing is the 
transmission of information over a noisy channel. This 
central paradigm of information theory can be mathe- 
matically abstracted to having two random variables X 
and Y representing the desired information and its noisy 
replica, respectively. Noisy transmission can occur either 
via space from one geographical point to another, as hap- 
pens in communications, or in time, for example, when 
writing or reading files from a hard disk in the computer. 

The most important measures of information are en- 
tropy and mutual information. Information (Shannon) 
entropy, H(-), is a measure of the amount of uncertainty 



in a random variable, indicating how easily data can be 
compressed. Mutual information quantifies the amount 
of information in common between two random variables 
and it is used to upper bound the attainable rate of infor- 
mation transferred across a channel. To put differently, 
mutual information, I(X; Y), measures the amount of in- 
formation that can be obtained about one random vari- 
able (the channel input X) by observing another (the 
output Y) . A basic property of the mutual information 
is that I(X;Y) = H(X) - H(X\Y), hence knowing Y, 
we can save an average of I(X; Y) bits in encoding X 
compared to not knowing Y. 

The archetypal Gaussian channel is one of the most 
popular models in the field of informatics and it arises in 
numerous applications in information processing, mod- 
eling the relation between the latent input and the cor- 
rupted observed output [§]. Consider a real- valued chan- 
nel with input and output random variables X and Y, 
respectively, of the form 



Y = X + N, 



(1) 



where N ~ 7V(0, 1/snr) is a Gaussian noise independent 
of X, and snr > is the channel's signal-to-noise ratio 
(SNR) . The input is taken from a probability distribution 
P{X) with a bounded second moment. In this Letter, 
random variables are denoted by upper case letters and 
their values denoted by lower case letters. 

The Gaussian channel ([1]) can be also viewed as a phys- 
ical system 0, [l(J EH , operating under the laws of ther- 
modynamics. The microstates of the thermal system are 
equivalent to the hidden values of the input X. A com- 
parison of the channel's a-posteriori probability distribu- 
tion, given by B ayes' law 



P(X = x\Y = y) 
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with the Boltzmann distribution law yields the following 
mapping of the inverse temperature and energy of the 
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equivalent thermal system 
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S(X = x\Y = y;f3).(2) 



In order to capture the temperature-dependent nature 
of the energy in systems like the communication channel, 
we generalize the formulation of the second law of ther- 
modynamics dS = dQ/T. The differential of the parti- 
tion function's logarithm, logZ, can be written as 

dlogZ 
dlo g Z = — [r dp. 

Utilizing the identity logZ = — (3U + S, one obtains 

dS = difiU) + 



dp 



Since for T-dependent energy 
d log Z 



dp 



we get 



dS = d(BU) -Ud/3- p(^^l)dp 

\ dp I 

Recalling that according to the first law dll = dQ con- 
cludes the generalized second thermodynamic law 



T T\ dT I 



(3) 



The generalized second law of thermodynamics has 
a clear physical interpretation. For simplicity, let us as- 
sume that an examined system is characterized by a comb 
of discrete energy levels SI, £2, .... The heat absorbed 
into the T-dependent system has the following dual effect: 
A first contribution of the heat, dU— < d£{X)/dT > dT, 
increases the temperature of the system while the sec- 
ond contribution, < d£{X)/dT > dT, goes for shifting 
the energy comb. However, the shift of the energy 
comb does not affect the entropy, since the occupation 
of each energy level remains the same, and the entropy 
is independent of the energy values which stand behind 

the labels £1,£2, The change in the entropy can 

be done only by moving part of the occupation of one 
tooth of the energy comb to the neighboring teeth, and 
this can be achieved only by changing the temperature. 
Hence, the effective heat contributing to the entropy is 
dQ— < d£{X)/dT > dT, and this is the physical expla- 
nation to the generalized second law (J3|. A schematic 
picture of communication-heat-engine is depicted in Fig- 
ure 1, where the heat devoted to the change of the Hamil- 
tonian is denoted by the term 'working channel'. 
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Figure 1: A schematic communication-heat-engine. H/C de- 
note Hot/Cold temperatures. 



Note that the first law remains unaffected, dU = dQ, 
since both ways of heat flow absorption into the system 
are eventually contributing to the average internal energy 
U . The generalized second law specifies the trajectory, 
the weight of each one of the two possible heat flows, at 
a given temperature. 

The temperature, originally defined by the second law 
as T = dQ/dS, is redefined now as 



dS/dQ 



1 



J< , _ «l£(X)/dT> 
dQ/dT 



dS/dQ 



1 



<d£(X)/dT> 
C V {T) 



(4) 



This redefinition has a more complex form and involves 
an implicit function of T, since the temperature appears 
on both sides of the equation. Note that the form of the 
denominator of (j4]) resembles an efficiency factor of the 
communication-heat-engine, and will be discussed at the 
concluding paragraphs. 

Based on the generalized second law, a thermody- 
namic expression for the mutual information of quasi- 
static communication channels can be derived. To make 
the derivation intelligible we start with the thermody- 
namic expression for the mutual information in the case 
of a temperature-independent Hamiltonian. 

The entropy integral obtained from combining the 
three laws of thermodynamics for the thermal system be- 
ing equivalent to the Gaussian channel, is 



S(p) = S(X\Y = y;p) 



dmy = yyy) 

7 ~ A dry, 
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where the internal energy, 

U(Y =y;P) =E X{Y {£(X\Y =y)}, is the energy 
averaged over all possible values of X, given y. In this 
contribution, the symbol Ex{-} denotes expectation 
of the random object within the brackets with respect 
to the subscript random variable. The posterior in- 
formation (Shannon) entropy, H(X\Y; p), (in nats) of 
the channel can be expressed via the thermodynamic 
entropy conditioned on Y = y, S(X\Y = y; P), as 



H(X\Y;p) = E Y {S(X\Y = y;P)} 

dU(Y;j) 
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The input's entropy can also be reformulated in a similar 
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manner, since H(X) = H(X\Y; (3 = 0). Hence, 

dU(Y; 7 ) d ^ 

Now, the input-output mutual information can be de- 
scribed via thermodynamic quantities, namely the en- 
ergy, £ , and inverse temperature, (3, as 

I(X;Y)=I([3)=H(X)-H(X\Y;f3) 





dU(Y ;l ), 



= -[ 1 E Y {U(Y; 1 )}f Q +E Y { I U(Y ]1 )d 1 



(5) 



For the generalized second thermodynamic law one has 
to insert the effective heat contribution to the entropy 
and the mutual information is given by 



I(X;Y) = -[ 1 E Y {U(Y; 1 )}] Q + 
E Y {1\U(Y; 7)+ ,E XIY {^^ 



(6) 
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Note that this thermodynamic expression for the mutual 
information holds for any channel which can be described 
by a thermal system exhibiting quasi-static heat transfer. 

For the Gaussian channel with a standard Gaus- 
sian input, 7V(0,1), we get log(P(X))/(3 = ~x 2 /(2/3) 
and the energy ^ of the Gaussian channel sys- 
tem becomes an explicit function of /3, given 
by £(X = x\Y = y;(3) = -xy + x 2 (l + {3)/2(3. 

The derivative of this function with respect to 
(3 yields d£(X = x\Y = y; (3)/d(3 = -x 2 /{2(3 2 ). 
The a-posteriori probability density function is 
p(X = x\Y = y;/3) = N((3y/(\ + (3), 1/(1 + /?)). Hence, 
the internal energy is 



U(Y = y;/3)= E X]Y {£ (X\Y = y;p)} = - 



V 2 (3 



1 
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The derivative of the energy averaged over all possible 
inputs is 



E 



X\Y 



\ d£{X\Y = y;l3) \ _ 1 
1 } 
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and the marginal probability density function of the out- 
put is given by p(Y = y) = A/"(0, (1 + 13)/ (3). Incorporat- 
ing all these results into the mutual information expres- 
sion |(6j) one can easily show that 

I(X;Y) = i log + 

and the celebrated formula for the Shannon capacity Q 
is derived from the perspective of thermodynamics. 



For the Gaussian channel with Bernoulli- 1/2 input, 
i.e. P(X = 1) = P(X = -1) = 1/2, the X 2 /2 = 1/2 
and \ogP(X = x)/(3 terms of the Gaussian channel's 
energy <[2j) are independent of X and can be canceled 
out by the same terms coming from the partition 
function, leaving us with the expression, independent 
of (3, £(X = x\Y = y) = —xy. The internal energy 
is U(Y = y;(3) = E X]Y {£ (X\Y = y)} = -y tardily) 
and the marginal probability density function 
of the output is then given by p(Y = y) = 
(^/(2V2^))[exp(-{3(y - l) 2 /2) + exp(-(3(y + l) 2 /2)]. 
Incorporating these definitions into eq. <[5j) for the case of 
energy function which is independent of the temperature 
one can verify that 



1(13) = f3- 



1 



cxp 



2 



log cosh (13- y r ]3y)dy, 



which is identical to the known Shannon-theoretic result 
(see, e.g., [H, eq. (18)] and [H p. 274]). 

We turn now to a discussion on the physical implica- 
tion of the generalized second thermodynamic law and 
the redefinition of the temperature. 

At equilibrium, a system with a T-dependent Hamil- 
tonian is occupying the macroscopic state which maxi- 
mizes the entropy. We argue that such generalized sys- 
tems obey the following traditional physical picture: (a) 
Two systems which are in thermal contact and at equi- 
librium have the same (redefined) temperature, (b) For 
two systems which are in thermal contact and at gen- 
eralized temperature T\ and T2, heat always flows from 
high temperature to low temperature. In addition to this 
traditional picture, (c) systems fulfilling < d£/dT > > 
achieve an equilibrium state, whereas systems obeying 

< d£/dT > < are unstable to thermal fluctuations, 
hence cannot describe a physical system at equilibrium. 

For generalized systems, the redefined inverse temper- 
ature is 1/T = dS/[dQ- < d£/dT > dT}. Hence, (3 has 
the same meaning as for the classical second law but with 
an effective heat contributing to the entropy. When two 
systems are in thermal contact the total change in the 
entropy is dS = dSi + dS2 ■ A necessary condition for a 
maximum of the total entropy, dS, is that the slope of 
the entropy with an effective heat is the same for both 
systems. Otherwise, heat will flow from a system with 
the lower slope to the one with the higher slope and the 
total entropy will increase. Hence it is clear that at a 
thermal equilibrium the redefined temperatures of the 
two systems have to obey T\ = T 2 as for the conventional 
thermodynamic picture. 

We present now a self-consistent argument to prove 
that a generalized system is unstable in the event of 

< d£ /dT > < 0. Assume a generalized system is in ther- 
mal equilibrium with a reservoir, and time-dependent 
fluctuations (of heat transfers) between the measured 
system and the reservoir. Fluctuations that increase 
(decrease) the internal energy of the generalized system 
move the energy comb to 'right' ('left'), such that a higher 
(lower) energy is assigned to each one of the teeth of the 
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comb. For any macroscopic state there is a degeneracy, 
i.e., many microscopic states obeying the same macro- 
scopic parameters. We assume that the degeneracy of 
the system increases in a move to the right tooth, corre- 
sponding to a relatively higher energy level. 

For the simplicity of the discussion below, assume that 
the generalized system is always occupying one tooth of 
the energy comb only. In an event where the system ab- 
sorbs some heat from the reservoir by fluctuations, there 
are two 'options' for the system how to 'invest' the ab- 
sorbing heat. The first option is to move the energy comb 
to the 'right', and the second one is to keep the current 
position of the energy comb, but to move the system to 
a higher (right position) tooth. The first option does 
not change the entropy, while the second one increases 
the entropy. Hence, the system adopts the second op- 
tion. Similarly when the system omits heat to the reser- 
voir the system prefers to move the energy comb to the 
'left', instead of moving to a lower tooth. Such processes 
will repeat again and again, until the energy comb sat- 
urates the leftmost position, and the system moves to 
the highest-energy tooth, independent of the tempera- 
ture of the reservoir. When the 'left' move of the en- 
ergy comb is unbounded, the process will not terminate. 
Note that indeed for the Gaussian channel (see eq. 2) 

< d£/dT > = < x 2 /2 > > 0, and the system achieves an 
equilibrium state. 

Let us examine now two systems at T\ < T 2 , 

< d£i/dTi > > and < d£ 2 /dT 2 > > 0. Assume self- 
consistently that the first (second) system absorbs 
(emits) heat. The total change in the entropy can be 
expressed as 

<d S = dS 1 + dS 2 = ^-{l-^) 

dl 1 

where it is clear that dT x > and dT 2 < 0. (If dT x < 
then the first system absorbs more heat than dQ, i.e., 
dQ— < d£\jdT\ > dT\.) Now it can be easily verified 
that the last term (second line) of the r.h.s of |(7]) is neg- 
ative. Hence, a necessary condition for the total entropy, 



dS, to be positive is that the first term of the r.h.s. of 
([7]) is positive, T 2 > Ti, implying that heat flows from 
high to low temperatures. 

T-dependent (independent) Hamiltonians arise also for 
discrete (continuous) input symbols. An example for 
such T-dependent Hamiltonians is the Gaussian chan- 
nels with non-equiprobable binary inputs, T(±l) 7^ 1/2, 
whereas an example for a T-independent Hamiltonian 
is the Gaussian channel with a uniform input distribu- 
tion. It is clear that the characterization of communica- 
tion channels with T-dependent /independent Hamiltoni- 
ans does not follow the type of the input /output symbols. 
What is the origin, from the point of view of physics, for 
the T-dependent Hamiltonians? In order to answer this 
question one has to examine a possible conflict between 
the nature of the heat and the constraint imposed by the 
communication channel. 

For a communication channel governed by a T- 
dependent Hamiltonian, e.g. eq. (2), the distribution of a 
microscopic state exp(—f3H) oc P(X) which is indepen- 
dent of (3. This is a nonphysical postulate assigned to 
the hidden values of the input X, a degree of freedom 
in the physical system. The distribution of a degree of 
freedom (e.g., spin) governed by a Hamiltonian system 
with/without an inversion symmetry can be nonuniform, 
but it always depends on the temperature. There is a 
natural distribution inspired by the heat and a distribu- 
tion imposed by the communication channel. In case of 
a conflict between these two distributions the constraint 
imposed by the channel wins, and the system has to de- 
vote some heat to implement it. A heat 'piston' is re- 
quired to bend the natural heat-distribution to that of 
the channel, and this is the origin for the working chan- 
nel by the communication-heat-engine (Figure 1). 

The non-prefect communication-heat-engine has also 
a meaningful fingerprint in the additional (third term) 
of the generalized mutual information, eq. (6), which 
is negative since < d£/df3 > < 0, (< d£/dT > > 0). It 
indicates that the waste of heat playing against the laws 
of nature decreases the mutual information in comparison 
to the same model (communication channel) with the 
lack of the T-dependent term in the Hamiltonian. 
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